Not So Randomly Typing Monkeys – Rank-frequency Behavior of Natural and Artificial Languages

Power laws arise through many natural processes. Zipf showed that the frequencies of words, as they appear in Shakespeare’s Hamlet, follow a power law distribution. Mandelbrot explained this effect as a result of an underlying information-theoretic optimization problem. Miller invoked doubt by showing that a very simple mechanism could also explain the presence of power laws: A monkey typing words with uniformly and independently selected letters would also produce word frequencies following a power law. In consequence, several other researchers proposed and investigated rankfrequency distributions of randomly generated text. In this paper, we first present a literature overview over this exciting topic. We then propose a class of Hidden Markov Models (HMMs) which generalizes the models previously investigated, generating power law, log-normal and other behavior. We extend a result of Conrad and Mitzenmacher for computing the power law exponent of zero order Markov processes to a setting which captures random walks in d-regular graphs. In an extensive empirical evaluation, we investigate convergence of rank frequency distributions for randomly generated text to those of natural language corpora, for increasing orders of Markov Processes and HMMs with an increasing number of hidden states. Our analysis uses four real-world corpora: the Reuters corpus, Shakespeare’s Hamlet, Goethe’s Faust and source code of the Linux kernel.